Introduction

This report was created by R version 4.2.3 and its package ShinyItemAnalysis version 1.5.0. ShinyItemAnalysis provides test and item analysis and it is available on CRAN and also online.

To cite ShinyItemAnalysis application in publications, please, use:

Martinkova P., & Drabinova A. (2018) ShinyItemAnalysis for teaching psychometrics and to enforce routine analysis of educational tests. The R Journal, 10(2), 503-515. doi: 10.32614/RJ-2018-074

This program is free software and you can redistribute it and or modify it under the terms of the GNU GPL 3 as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability of fitness for a particular purpose. Visit www.ShinyItemAnalysis.org to learn more!

Summary

Total scores

Summary table of total scores

The table below summarizes basic characteristics of total scores including number of respondents, minimum and maximum values, mean, median, standard deviation, skewness, and kurtosis. The skewness for normally distributed scores is near the value of 0 and the kurtosis is near the value of 3.

n nc Min Max Mean Median SD Skewness Kurtosis
39775 39775 42 168 100.27 100 30.03 0 2

Histogram of total scores

For cut-score 100, the blue part of a histogram shows respondents with a total score above the cut-score, the grey column shows respondents with a total score equal to the cut-score and the red part of a histogram shows respondents below the cut-score.

Scoring

Summary table of standard scores

The total score, also known as the raw score, is the total number of correct answers. It can be used to compare an individual score to a norm group, e.g. if the mean is 12, then an individual score can be compared to see if it is below or above this average. The percentile indicates the value below which a percentage of observations falls, e.g. an individual score at the 80th percentile means that the individual score is the same or higher than the scores of 80% of all respondents. The success rate is the percentage of correct answers, e.g. if the maximum points of a test is equal to 20 and an individual score is 12 then the success rate is 12/20 = 0.6, i.e. 60%. The Z-score, or the standardized score, is a linear transformation of the total score with a mean of 0 and with a variance of 1. If X is the total score, M is its mean and SD is its standard deviation then Z-score = (X - M) / SD. The T-score is the transformed Z-score with a mean of 50 and a standard deviation of 10. If Z is Z-score then T-score = (Z * 10) + 50.

Total score Percentile Success rate Z-score T-score
42 0.00 25.00 -1.94 30.60
43 0.01 25.60 -1.91 30.93
44 0.01 26.19 -1.87 31.26
45 0.01 26.79 -1.84 31.59
46 0.02 27.38 -1.81 31.93
47 0.02 27.98 -1.77 32.26
48 0.03 28.57 -1.74 32.59
49 0.03 29.17 -1.71 32.93
50 0.04 29.76 -1.67 33.26
51 0.04 30.36 -1.64 33.59
52 0.05 30.95 -1.61 33.93
53 0.05 31.55 -1.57 34.26
54 0.06 32.14 -1.54 34.59
55 0.07 32.74 -1.51 34.92
56 0.07 33.33 -1.47 35.26
57 0.08 33.93 -1.44 35.59
58 0.09 34.52 -1.41 35.92
59 0.10 35.12 -1.37 36.26
60 0.11 35.71 -1.34 36.59
61 0.11 36.31 -1.31 36.92
62 0.12 36.90 -1.27 37.26
63 0.13 37.50 -1.24 37.59
64 0.14 38.10 -1.21 37.92
65 0.15 38.69 -1.17 38.25
66 0.16 39.29 -1.14 38.59
67 0.17 39.88 -1.11 38.92
68 0.17 40.48 -1.07 39.25
69 0.18 41.07 -1.04 39.59
70 0.19 41.67 -1.01 39.92
71 0.20 42.26 -0.97 40.25
72 0.21 42.86 -0.94 40.59
73 0.22 43.45 -0.91 40.92
74 0.23 44.05 -0.87 41.25
75 0.24 44.64 -0.84 41.59
76 0.25 45.24 -0.81 41.92
77 0.26 45.83 -0.77 42.25
78 0.27 46.43 -0.74 42.58
79 0.28 47.02 -0.71 42.92
80 0.29 47.62 -0.67 43.25
81 0.30 48.21 -0.64 43.58
82 0.31 48.81 -0.61 43.92
83 0.32 49.40 -0.58 44.25
84 0.33 50.00 -0.54 44.58
85 0.34 50.60 -0.51 44.92
86 0.35 51.19 -0.48 45.25
87 0.37 51.79 -0.44 45.58
88 0.38 52.38 -0.41 45.91
89 0.39 52.98 -0.38 46.25
90 0.40 53.57 -0.34 46.58
91 0.41 54.17 -0.31 46.91
92 0.42 54.76 -0.28 47.25
93 0.43 55.36 -0.24 47.58
94 0.44 55.95 -0.21 47.91
95 0.46 56.55 -0.18 48.25
96 0.46 57.14 -0.14 48.58
97 0.48 57.74 -0.11 48.91
98 0.49 58.33 -0.08 49.24
99 0.50 58.93 -0.04 49.58
100 0.51 59.52 -0.01 49.91
101 0.52 60.12 0.02 50.24
102 0.53 60.71 0.06 50.58
103 0.54 61.31 0.09 50.91
104 0.55 61.90 0.12 51.24
105 0.57 62.50 0.16 51.58
106 0.58 63.10 0.19 51.91
107 0.59 63.69 0.22 52.24
108 0.60 64.29 0.26 52.57
109 0.61 64.88 0.29 52.91
110 0.62 65.48 0.32 53.24
111 0.63 66.07 0.36 53.57
112 0.64 66.67 0.39 53.91
113 0.65 67.26 0.42 54.24
114 0.67 67.86 0.46 54.57
115 0.68 68.45 0.49 54.91
116 0.69 69.05 0.52 55.24
117 0.70 69.64 0.56 55.57
118 0.71 70.24 0.59 55.90
119 0.72 70.83 0.62 56.24
120 0.73 71.43 0.66 56.57
121 0.74 72.02 0.69 56.90
122 0.75 72.62 0.72 57.24
123 0.76 73.21 0.76 57.57
124 0.77 73.81 0.79 57.90
125 0.78 74.40 0.82 58.24
126 0.79 75.00 0.86 58.57
127 0.79 75.60 0.89 58.90
128 0.80 76.19 0.92 59.23
129 0.81 76.79 0.96 59.57
130 0.82 77.38 0.99 59.90
131 0.83 77.98 1.02 60.23
132 0.84 78.57 1.06 60.57
133 0.84 79.17 1.09 60.90
134 0.85 79.76 1.12 61.23
135 0.86 80.36 1.16 61.57
136 0.87 80.95 1.19 61.90
137 0.87 81.55 1.22 62.23
138 0.88 82.14 1.26 62.57
139 0.89 82.74 1.29 62.90
140 0.89 83.33 1.32 63.23
141 0.90 83.93 1.36 63.56
142 0.91 84.52 1.39 63.90
143 0.91 85.12 1.42 64.23
144 0.92 85.71 1.46 64.56
145 0.92 86.31 1.49 64.90
146 0.93 86.90 1.52 65.23
147 0.93 87.50 1.56 65.56
148 0.94 88.10 1.59 65.90
149 0.94 88.69 1.62 66.23
150 0.95 89.29 1.66 66.56
151 0.95 89.88 1.69 66.89
152 0.96 90.48 1.72 67.23
153 0.96 91.07 1.76 67.56
154 0.96 91.67 1.79 67.89
155 0.97 92.26 1.82 68.23
156 0.97 92.86 1.86 68.56
157 0.97 93.45 1.89 68.89
158 0.98 94.05 1.92 69.23
159 0.98 94.64 1.96 69.56
160 0.98 95.24 1.99 69.89
161 0.99 95.83 2.02 70.22
162 0.99 96.43 2.06 70.56
163 0.99 97.02 2.09 70.89
164 0.99 97.62 2.12 71.22
165 0.99 98.21 2.16 71.56
166 1.00 98.81 2.19 71.89
167 1.00 99.40 2.22 72.22
168 1.00 100.00 2.26 72.56

More complex estimates of ability are provided in the IRT section.

Validity

Correlation structure

Correlation heat map

A correlation heat map displays Pearson correlations of items. Pearson correlation coefficient describes linear correlation between two random variables X and Y.

Predictive validity

An association between the total score and the criterion variable can be estimated using Pearson product-moment correlation coefficient r. The null hypothesis being tested states that correlation is exactly 0.

r(39773) = -.04, p = <.001, 95% CI [-.05, -.03]

Interpretation:
The p-value is less than .05, thus we reject the null hypotheses. The total score and criterion variable are negatively correlated.

Traditional item analysis

Item analysis

Difficulty/Discrimination plot

Difficulty (red) of items is estimated as a percent of the respondents who answered correctly to that item. Discrimination (blue) is described by Pearson correlation between an item and the total score.

Cronbach’s alpha

Cronbach’s alpha is an estimate of internal consistency of a psychometric test. It is a function of the number of items in a test, the average covariance between item-pairs, and the variance of the total score (Cronbach, 1951).

Estimate Confidence interval
0.970 (0.969, 0.970)

Traditional item analysis table

Diff. - difficulty of the item is estimated as its average score divided by its range, Avg. score - average score of the item, SD - standard deviation, ULI - Upper-Lower Index calculated as the difference in the percent correct between the upper and lower third of respondents, RIT - Pearson correlation between an item and the total score, RIR - Pearson correlation between an item and the rest of the items, Alpha Drop - Cronbach's alpha of a test without a given item.
Item Diff. Avg. score SD ULI RIT RIR Alpha Drop
X.Q1A. 0.54 2.62 1.03 0.56 0.71 0.69 0.97
X.Q2A. 0.39 2.17 1.11 0.40 0.49 0.46 0.97
X.Q3A. 0.41 2.23 1.04 0.56 0.72 0.70 0.97
X.Q4A. 0.32 1.95 1.04 0.48 0.62 0.60 0.97
X.Q5A. 0.51 2.52 1.07 0.58 0.71 0.69 0.97
X.Q6A. 0.51 2.54 1.05 0.51 0.65 0.62 0.97
X.Q7A. 0.31 1.92 1.03 0.48 0.63 0.60 0.97
X.Q8A. 0.49 2.48 1.05 0.56 0.70 0.69 0.97
X.Q9A. 0.56 2.67 1.07 0.54 0.66 0.64 0.97
X.Q10A. 0.48 2.45 1.14 0.63 0.72 0.70 0.97
X.Q11A. 0.60 2.80 1.05 0.58 0.72 0.70 0.97
X.Q12A. 0.48 2.43 1.07 0.56 0.69 0.68 0.97
X.Q13A. 0.59 2.78 1.07 0.65 0.77 0.75 0.97
X.Q14A. 0.53 2.58 1.08 0.43 0.54 0.51 0.97
X.Q15A. 0.28 1.83 0.99 0.43 0.60 0.58 0.97
X.Q16A. 0.51 2.52 1.11 0.61 0.71 0.69 0.97
X.Q17A. 0.55 2.66 1.16 0.66 0.74 0.72 0.97
X.Q18A. 0.49 2.48 1.07 0.46 0.57 0.55 0.97
X.Q19A. 0.32 1.95 1.07 0.41 0.52 0.49 0.97
X.Q20A. 0.44 2.32 1.12 0.60 0.70 0.68 0.97
X.Q21A. 0.45 2.35 1.17 0.67 0.74 0.72 0.97
X.Q22A. 0.45 2.34 1.03 0.54 0.69 0.68 0.97
X.Q23A. 0.19 1.56 0.86 0.33 0.54 0.52 0.97
X.Q24A. 0.48 2.44 1.05 0.57 0.71 0.69 0.97
X.Q25A. 0.39 2.18 1.08 0.47 0.59 0.56 0.97
X.Q26A. 0.55 2.66 1.07 0.61 0.74 0.72 0.97
X.Q27A. 0.54 2.61 1.05 0.56 0.69 0.68 0.97
X.Q28A. 0.41 2.22 1.07 0.57 0.71 0.69 0.97
X.Q29A. 0.55 2.65 1.06 0.58 0.71 0.69 0.97
X.Q30A. 0.46 2.39 1.08 0.54 0.66 0.64 0.97
X.Q31A. 0.46 2.38 1.04 0.54 0.69 0.67 0.97
X.Q32A. 0.48 2.45 1.02 0.48 0.63 0.61 0.97
X.Q33A. 0.47 2.41 1.05 0.56 0.71 0.69 0.97
X.Q34A. 0.54 2.63 1.15 0.67 0.74 0.73 0.97
X.Q35A. 0.43 2.30 1.00 0.47 0.63 0.61 0.97
X.Q36A. 0.42 2.27 1.11 0.61 0.72 0.70 0.97
X.Q37A. 0.46 2.37 1.14 0.61 0.69 0.67 0.97
X.Q38A. 0.46 2.39 1.19 0.67 0.73 0.71 0.97
X.Q39A. 0.48 2.45 1.02 0.53 0.69 0.68 0.97
X.Q40A. 0.55 2.65 1.11 0.56 0.66 0.64 0.97
X.Q41A. 0.32 1.97 1.05 0.46 0.60 0.57 0.97
X.Q42A. 0.56 2.68 1.03 0.49 0.63 0.61 0.97

Distractor analysis

Distractor plot

Respondents are divided into a selected number of groups by their total score. Subsequently, the percentage of respondents in each group who selected a given answer (correct answer or distractor) is displayed. The correct answer should be selected more often by the respondents with a higher total score than by those with a lower total score, i.e. the solid line should be increasing. The distractor should work in the opposite direction, i.e. the dotted lines should be decreasing.

Multinomial plot

In the multinomial plot, points represent proportion of selected option with respect to total score. Their size is determined by count of respondents who achieved given level of total score and who selected given option.

IRT models

Item Response Theory (IRT) models are mixed-effect regression models in which the respondent’s ability \(\theta\) is assumed to be a latent and is estimated together with item parameters.

Wright (item-person) map using 1PL IRT model

A Wright map, also called an item-person map, is a graphical tool to display person estimates and item parameters. The person side (left) represents a histogram of estimated knowledge of the respondents. The item side (right) displays estimates of the difficulty of particular items.

Equation

All subsequent analyses are based on the selected 2PL IRT model: \[\mathrm{P}(Y_{pi} = 1|\theta_p) = \pi_{pi} = \frac{e^{a_i(\theta_p - b_i)}}{1 + e^{a_i(\theta_p - b_i)}}\] Model parameters are estimated using a marginal maximum likelihood method. Ability \(\theta\) is assumed to follow standard normal distribution.

Ability estimates

Ability is estimated using an EAP algorithm and a 2PL IRT model. The relationship between ability estimates (factor scores, F-scores) and standardized total test scores (Z-scores) can be seen on the scatter plot below. A table with ability estimates for all respondents can be downloaded from the application.

Score type Min Max Mean Median SD Skewness Kurtosis
Total Scores Total Scores 42.00 168.00 100.27 100.00 30.03 0.12 2.17
Z-Scores Z-Scores -1.94 2.26 0.00 -0.01 1.00 0.12 2.17
F-scores F-Scores -1.28 2.92 0.00 0.09 0.94 0.15 2.30

Item characteristic and information curves

Parameter estimates and item fit

Estimates of parameters are completed by their standard errors (SE) and by signed Chi-squared statistics S-X2 (see Orlando and Thissen, 2000). P-values lower than 0.05 indicate suspicious items not fitting the selected IRT model. S-X2 is computed only when no missing data are present.

a SE(a) b SE(b) c SE(c) d SE(d) S-X2 df p-value
X.Q1A. 2.45 0.03 0.80 0.01 0 NA 1 NA 46.57 38 0.16
X.Q2A. 1.25 0.02 1.51 0.02 0 NA 1 NA 79.48 39 0.00
X.Q3A. 2.70 0.04 1.16 0.01 0 NA 1 NA 109.83 38 0.00
X.Q4A. 1.88 0.03 1.56 0.02 0 NA 1 NA 51.65 39 0.08
X.Q5A. 2.42 0.03 0.86 0.01 0 NA 1 NA 65.94 38 0.00
X.Q6A. 2.00 0.03 0.93 0.01 0 NA 1 NA 67.78 39 0.00
X.Q7A. 2.00 0.03 1.54 0.02 0 NA 1 NA 60.82 39 0.01
X.Q8A. 2.25 0.03 0.96 0.01 0 NA 1 NA 54.96 38 0.04
X.Q9A. 1.83 0.02 0.76 0.01 0 NA 1 NA 35.59 39 0.63
X.Q10A. 2.68 0.04 0.79 0.01 0 NA 1 NA 93.21 38 0.00
X.Q11A. 2.43 0.03 0.54 0.01 0 NA 1 NA 49.54 37 0.08
X.Q12A. 2.22 0.03 1.02 0.01 0 NA 1 NA 49.16 39 0.13
X.Q13A. 3.20 0.04 0.49 0.01 0 NA 1 NA 74.30 36 0.00
X.Q14A. 1.41 0.02 0.96 0.01 0 NA 1 NA 34.68 39 0.67
X.Q15A. 2.25 0.04 1.62 0.02 0 NA 1 NA 42.81 39 0.31
X.Q16A. 2.47 0.03 0.79 0.01 0 NA 1 NA 54.17 38 0.04
X.Q17A. 2.78 0.03 0.53 0.01 0 NA 1 NA 98.11 37 0.00
X.Q18A. 1.70 0.02 1.06 0.01 0 NA 1 NA 126.21 39 0.00
X.Q19A. 1.33 0.02 1.77 0.02 0 NA 1 NA 54.98 39 0.05
X.Q20A. 2.36 0.03 0.99 0.01 0 NA 1 NA 50.76 38 0.08
X.Q21A. 3.16 0.04 0.80 0.01 0 NA 1 NA 158.15 37 0.00
X.Q22A. 2.29 0.03 1.14 0.01 0 NA 1 NA 65.37 39 0.01
X.Q23A. 2.03 0.04 2.08 0.02 0 NA 1 NA 62.17 39 0.01
X.Q24A. 2.42 0.03 0.98 0.01 0 NA 1 NA 56.67 38 0.03
X.Q25A. 1.69 0.03 1.38 0.02 0 NA 1 NA 52.12 39 0.08
X.Q26A. 2.84 0.04 0.68 0.01 0 NA 1 NA 107.89 37 0.00
X.Q27A. 2.24 0.03 0.80 0.01 0 NA 1 NA 45.59 38 0.19
X.Q28A. 2.42 0.03 1.16 0.01 0 NA 1 NA 63.12 39 0.01
X.Q29A. 2.24 0.03 0.74 0.01 0 NA 1 NA 43.02 38 0.27
X.Q30A. 1.98 0.03 1.07 0.01 0 NA 1 NA 54.92 39 0.05
X.Q31A. 2.29 0.03 1.08 0.01 0 NA 1 NA 62.07 39 0.01
X.Q32A. 1.90 0.03 1.14 0.01 0 NA 1 NA 45.90 39 0.21
X.Q33A. 2.35 0.03 1.04 0.01 0 NA 1 NA 48.74 38 0.11
X.Q34A. 2.94 0.04 0.56 0.01 0 NA 1 NA 111.10 36 0.00
X.Q35A. 2.00 0.03 1.32 0.01 0 NA 1 NA 45.10 39 0.23
X.Q36A. 2.74 0.04 1.00 0.01 0 NA 1 NA 56.81 38 0.03
X.Q37A. 2.38 0.03 0.88 0.01 0 NA 1 NA 105.25 38 0.00
X.Q38A. 2.87 0.04 0.74 0.01 0 NA 1 NA 107.68 37 0.00
X.Q39A. 2.36 0.03 1.04 0.01 0 NA 1 NA 45.72 38 0.18
X.Q40A. 1.93 0.02 0.70 0.01 0 NA 1 NA 56.23 38 0.03
X.Q41A. 1.76 0.03 1.57 0.02 0 NA 1 NA 67.00 39 0.00
X.Q42A. 1.78 0.02 0.81 0.01 0 NA 1 NA 111.19 39 0.00